Computational notebook for PhD Seminar

Statistical method of averaging

• Consider the simplistic but most popular measure of averaging
• Suppose we observe a set of the price change path phenomenon of a stock over t periods \(x_1, x_2,...,x_t\).

[1] 112.2274

• The averaging function highlighted above is an algorithm and is represented by the red line in the plot opposite

Fake Data and the average as an estimate

• The mean value summarises the path price into a single number
• But how accurate is this algorithm?
• The textbook answer is given in terms of the standard error

Standard error function

[1] 0.98525

• Here averaging is the .content-box-red[algorithm], while the standard error provides the .content-box-blue[inference] of the algorithm’s accuracy.

Explanation

• It is a surprising, and crucial, aspect of statistical theory that the same data that supplies an estimate can also assess its accuracy.
• Strictly speaking Inference concerns more than accuracy: recall that algorithms say what the statistician does while inference says why she does it.

Algorithms and Inference

• Of course, the standard_error() function defined previously is itself an algorithm, which could be (and is) subject to further inferential analysis concerning its accuracy

• .content-box-green[The point is that the algorithm comes first and the inference follows at a second level of statistical consideration.]

• .content-box-yellow[In practice this means that algorithmic invention is a more free-wheeling and adventurous enterprise]

• .content-box-grey[In contrast inference playing catch-up as it strives to assess the accuracy, good or bad, of some hot new algorithmic methodology.]

Algorithms and regression

.panelset[ .panel[.panel-name[Least squares algorithm for linear regression] - The least squares estimator is a popular algorithm for estimating a linear regression - The algorithm fits the data by least squares, by minimising the sum of squared deviations over all choices of the model parameters. - Consider the following fake relationship between the price and some market factor

Least squares algorithm

• This code manufactures a positive relationship to the factor plus some noise and then draws the least squares regression line.
• The accuracy of this estimate is given by \(\pm\) 2 standard errors
• The appropriate inference of this banded grey area is that this has a 95% chance in including the true expected value of Price in an Up market.
• This 95% coverage depends on the validity of the linear regression model, which could as easy have been a quadratic relationship

Lowess algorithm for localised regression

• Lowess is a modern computer based algorithm which works by moving its attention along the x-axis, fitting local polynomial curves of differing degrees to nearby (x,y) coordinates.
• The fitted estimate above has a similar linear regression as the least squares algorithm in the middle of the data but for higher values of the factor has a much steeper curve.

Hypothesis testing

• There has also been a march of methodology and inference for hypothesis testing rather than estimation
• Consider questions about where value investing is better in a bull or a bear market?
• The plots shows the histogram of these two groups in the uk data.Up markets to seem to have a more positive return then down markets.
• The mean values for up and not up are in fact 0.00039 and -0.00032 respectively
• Is the perceived difference genuine or as some people would say a statistical fluke

• The classic answer to this question is via a two-sample t-test \[\frac{\bar{Value_{UP}}-\bar{Value_{Not Up}}}{\hat{sd}}\]

• where \(\hat{sd}\) is estimate of the numerators standard deviation

• .content-box-blue[Dividing by sd allows us (under Gaussian assumptions) to compare the observed value of t with a standard null distribution, in this case a Student’s t distribution with 7388 degrees of freedom.]

Two sample t-test inference

• We obtain t= 5.07 which would classically be considered very strong evidence that the apparent difference is genuine; in standard terminology, “with two-sided significance level 0.0000003.”
• A small significance level (or “p-value”) is a statement of statistical surprise: something very unusual has happened if in fact there is no difference in returns of the value factor in up and down markets
• We are less surprised by t=5.07 if this comparison of mean returns is just one candidate out of thousands that might have produced “interesting” results.
• For example if I take a different set sample period and run the test again, or I split the sample into 5 sub-samples and run the test again.

Traditional hypothesis testing and false discovery theory

• A primary goal of empirical research is to identify the variables involved in a phenomenon
• The identification of these variables is a prerequisite to the formulation of a theory
• In classical statistics (e.g., Econometric), the significance of variables is established through p values
• p values suffer from multiple flaws, which have led to the acknowledgment that most discoveries in finance are false
• False discovery rate theory is an impressive advance in statistical inference, incorporating Bayesian ,frequentist, and empirical Bayesian elements.
• It was a necessary advance in a scientific world where computer-based technology routinely presents thousands of comparisons to be evaluated at once.

Simulated example of false discovery problem

set.seed(1235)
res<-tibble(testno=1:20, pvalue=1)
for (i in 1:20) {
  Up=rnorm(1000)
  NoUp=rnorm(1000)
  res[i,2]<-t.test(Up,NoUp)['p.value']
}

• This code fakes the return up versus down question earlier, where the ground truth is that there is no relationship.
• This implies that our significant results, when using a t-test, must be fake
• We repeat the test 20 times using a simple for loop.

• Arranging the results in ascending order of p-values.
• With enough hypothesis test we will always discovery a fake results.
• In financial machine learning, where we have to work with high-dimensional data with many features, 1000’s of comparisons are common.